10/512039 

DT01 Rec'd PCT/PTO 2 0 OCT 2004 

SPCIFIC ATION 
CAMERA CALIBRATING APPARATUS 

5 TECHNICAL FIELD OF THE INVENTION 

The present invention relates to a camera calibrating apparatus, and more 
particularly to a camera calibrating apparatus for calibrating a camera mounted on an 
automotive vehicle or the like. 

10 DESCRIPTION OF THE RELATED ART 

In recent years, there has been developed an imaging controlling apparatus, for 
example an electronic control unit (herein after simply referred to as "ECU"), which is used 
to be electrically connected to a camera mounted on an exterior surface of an automotive 
vehicle to detect the position occupied by an object located on a road, in particular, in the 
15 vicinity of the automotive vehicle on the basis of image information obtained by the camera. 
When, in general, the camera is mounted on the automotive vehicle in combination with the 
imaging controlling apparatus of this type, the calibration of the camera is performed by a 
camera calibrating apparatus in order to specify optical parameters of the camera. 

Up until now, there have been proposed a wide variety of camera calibrating 
20 apparatuses of this type, all of which are classified into two different types including a first 
type to calibrate the camera before that camera is mounted on the automotive vehicle, and a 
second type to calibrate the camera after that camera is mounted on the automotive vehicle. 
For the purpose of facilitating an assembly work of the camera to the automotive vehicle, 
there is a growing demand for the camera calibrating apparatus for calibrating the camera 
25 mounted on the automotive vehicle in recent years. 

As shown in FIGS. 39 to 41, the conventional camera calibrating apparatus 500 is 
electrically connected to the camera 510 as an imaging device. The camera 510 includes a 
housing unit 511 and an optical section 512 supported by the housing unit 511 to obtain 
image information through the optical section 512. 
30 The conventional camera calibrating apparatus 500 comprises a first housing 

position information storing section 515 for storing first housing position information 
indicative of the position occupied by the housing unit 511 in the first coordination system 
501 of a camera producing plant, and a second housing position information storing section 
516 for storing second housing position information indicative of the position occupied by 
35 the housing unit 511 in the second coordination system 502 of an automotive vehicle 
producing plant. 

The camera calibrating apparatus 500 is adapted to revise the camera 510 in the 
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camera producing plant by using a revising marker 505 located in the first coordination 
system 501. Here, the term "the calibration of the camera 510" is intended to indicate the 
calculation of the accurate position of the optical section 512 of the camera 510 mounted on 
the automotive vehicle 508. 
5 The conventional camera calibrating apparatus 500 further comprises a first optical 

position information producing section 517 for producing first optical position information 
indicative of the position of the optical section 512 in the first coordination system 501, and 
a first optical position information storing section 518 for storing the first optical position 
information produced by the first optical position information producing section 517. The 

10 first optical position information producing section 517 is adapted to produce optical 
position information indicative of the position of the optical section 512 in the first 
coordination system 501 on the basis of the image information of the revising marker 505 
obtained by the camera 510. 

The conventional camera calibrating apparatus 500 further comprises a second 

15 optical position information producing section 520 for producing second optical information 
indicative of the position of the optical section 512 in the second coordination system 502, 
and a second optical position information storing section 530 for storing the second optical 
position information produced by the second optical position information producing section 
520. 

20 The second optical position information producing section 520 is adapted to 

produce optical position information indicative of the position of the optical section 512 in 
the second coordination system 502 from the second housing position information stored in 
the second housing position information storing section 516 on the basis of the first housing 
position information stored in the first housing position information storing section 515 and 

25 the first optical position information stored in the first optical position information storing 
section 518. 

The conventional camera calibrating apparatus 500 thus constructed as previously 
mentioned is adapted to revise the camera in the camera producing plant by calculating the 
position of the optical section 512 in the second coordination system 502, while the image 
30 controlling apparatus is adapted to detect the position occupied by an object located on the 
road through the image information obtained by the camera 510 mounted on the automotive 
vehicle 508. 

The conventional camera calibrating apparatus as previously mentioned, however, 
encounters such a problem that the second optical position information stored in the second 
35 optical position information storing section is not calibrated. This leads to the fact that the 
position of the object located on the road cannot be accurately detected by the image 
controlling apparatus if the camera is mounted on an inaccurate position of the automotive 
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vehicle. 

It is, therefore, an object of the present invention to provide a camera calibrating 
apparatus which can calibrate optical parameters of the optical section of the camera 
mounted on the automotive vehicle or the like. 

5 

DISCLOSURE OF THE INVENTION 

In accordance with the present invention, there is provided a camera calibrating 
apparatus to be operative in combination with an imaging device which includes a housing 
unit and an optical section supported by the housing unit to obtain image information 

10 thorough the optical section, and adapted to calibrate position information of the optical 
section, comprising: first housing position information storing means for storing first 
housing position information indicative of the position of the housing unit in the first 
coordination system; second housing position information storing means for storing second 
housing position information indicative of the position of the housing unit in the second 

15 coordination system; first optical position information producing means for producing first 
optical position information indicative of the position of the optical section in the first 
coordination system on the basis of the image information obtained by the imaging device 
in the first coordination system; first optical position information storing means for storing 
the first optical position information produced by the first optical position information 

20 producing means; second optical position information producing means for producing 
second optical position information indicative of the position of the optical section in the 
second coordination system from the second housing position information stored by the 
second housing position information storing means on the basis of the first housing position 
information stored by the first housing position information storing means and the first 

25 optical position information stored by the first optical position information storing means; 
second optical position information storing means for storing the second optical position 
information produced by the second optical position information producing means; and 
calibrating means for calibrating the second optical position information stored by the 
second optical position information storing means on the basis of the image information 

30 obtained by the imaging device in the second coordination system. The camera calibrating 
apparatus according to the present invention thus constructed as previously mentioned can 
calibrate parameters of the optical section mounted on an automotive vehicle or the like. 

In accordance with the present invention, there is provided a camera calibrating 
apparatus to be operative in combination with an imaging device which includes a housing 

35 unit and an optical section supported by the housing unit to obtain image information 
thorough the optical section, and adapted to calibrate optical position information about the 
position of the optical section, comprising: optical position information storing means for 
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storing optical position information indicative of the position of the optical section to a 
predetermined coordinate system; and calibrating means for calibrating the optical position 
information stored by the optical position information storing means on the basis of the 
image information obtained by the imaging device on the predetermined coordinate system. 
5 The camera calibrating apparatus according to the present invention thus constructed as 
previously mentioned can calibrate parameters of the optical section mounted on an 
automotive vehicle or the like. 

In accordance with the present invention, there is provided a camera calibrating 
apparatus to be operative in combination with an imaging device which includes a housing 

10 unit and an optical section supported by the housing unit to obtain image information 
thorough the optical section, and adapted to calibrate optical position information about the 
position of the optical section, comprising: first housing position information storing means 
for storing first housing position information indicative of the position of the housing unit in 
the first coordination system in which a revising marker is located; second housing position 

15 information storing means for storing second housing position information indicative of the 
position of the housing unit in the second coordination system in which a calibrating marker 
is located; first optical position information producing means for producing first optical 
position information indicative of the position of the optical section in the first coordination 
system on the basis of the image information of the revising marker obtained by the imaging 

20 device; first optical position information storing means for storing the first optical position 
information produced by the first optical position information producing means; second 
optical position information producing means for producing second optical position 
information indicative of the position of the optical section in the second coordination 
system from the second housing position information stored by the second housing position 

25 information storing means on the basis of the first housing position information stored by 
the first housing position information storing means and the first optical position 
information stored by the first optical position information storing means; estimated location 
information producing means for producing estimated location information indicative of the 
position of the calibrating marker to an image coordination system of the imaging device on 

30 the basis of the second optical position information produced by the second optical position 
information producing means; second optical position information storing means for storing 
the second optical position information produced by the second optical position information 
producing means; estimated location information storing means for storing the estimated 
location information produced by the estimated location information estimating means; and 

35 calibrating means for calibrating the second optical position information stored by the 
second optical position information storing means on the basis of the image information of 
the calibrating marker obtained by the imaging device and the estimated location 
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information stored by the estimated location information storing means. The camera 
calibrating apparatus according to the present invention thus constructed as previously 
mentioned can calibrate the second optical position information by using a simple 
calibrating marker. 

5 In the camera calibrating apparatus according to the present invention, the 

calibrating means includes: image location information extracting means for extracting an 
image location information indicative of an image location of the calibrating marker to the 
image coordination system of the imaging device on the basis of the image information of 
the calibrating marker obtained by the imaging device; calibration value calculating means 

10 for calculating a calibration value of the second optical position information stored by the 
second optical position information storing means on the basis of the image location 
information extracted by the image location information extracting means and the estimated 
location information stored by the estimated location information storing means; and optical 
position information calibrating means for calibrating the second optical position 

15 information stored by the second optical position information storing means on the basis of 
the calibration value calculated by the calibration value calculating means. The camera 
calibrating apparatus according to the present invention thus constructed as previously 
mentioned can calculate calibration value of the second optical position information by 
using a simple calibrating marker. 

20 In the camera calibrating apparatus according to the present invention, the 

calibrating means is adapted to calibrate a deviation of a rotational component of the second 
optical position information. The camera calibrating apparatus according to the present 
invention thus constructed as previously mentioned can calibrate the second optical position 
information by using a simple calibrating marker without taking into account parallel shift 

25 component of the second optical position information. 

In the camera calibrating apparatus according to the present invention, the image 
location information extracting means includes: image displaying means for displaying an 
image of the calibrating marker obtained by the imaging device, and image location 
specifying means for specifying the image location of the calibrating marker on the image 

30 of the calibrating marker displayed by the image displaying means and to ensure that the 
image location information is extracted. The camera calibrating apparatus according to the 
present invention thus constructed as previously mentioned can specify the image location 
of the calibrating marker to ensure that the image location information of the calibrating 
marker is extracted. 

35 In the camera calibrating apparatus according to the present invention, the image 

location information extracting means includes: estimated area information storing means 
for storing estimated area information indicative of the calibrating marker to the image 
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coordination system of the imaging device; and image location searching means for 
searching the image location of the calibrating marker from the image information of the 
calibrating marker obtained by the imaging device on the basis of the estimated area 
information stored by the estimated area information storing means and the estimated 
5 location information stored by the estimated location information storing means to ensure 
that the image location information is extracted. The camera calibrating apparatus 
according to the present invention thus constructed as previously mentioned can search the 
image location of the calibrating marker to ensure that the image location information of the 
calibrating marker is easily extracted. 

10 In accordance with the present invention, there is provided a camera calibrating 

apparatus to be operative in combination with an imaging device which includes a housing 
unit and an optical section supported by the housing unit to obtain image information 
thorough the optical section, and adapted to calibrate optical position information about the 
position of the optical section, comprising: first housing position information storing means 

15 for storing first housing position information indicative of the position of the housing unit in 
the first coordination system in which a revising marker is located; second housing position 
information storing means for storing second housing position information indicative of the 
position of the housing unit in the second coordination system; first optical position 
information producing means for producing first optical position information indicative of 

20 the position of the optical section in the first coordination system on the basis of the image 
information obtained by the imaging device and indicative of the revising marker; first 
optical position information storing means for storing the first optical position information 
produced by the first optical position information producing means; second optical position 
information producing means for producing second optical position information indicative 

25 of the position of the optical section in the second coordination system from the second 
housing position information stored by the second housing position information storing 
means on the basis of the first housing position information stored by the first housing 
position information storing means and the first optical position information stored by the 
first optical position information storing means; second optical position information storing 

30 means for storing the second optical position information produced by the second optical 
position information producing means; and calibrating means for calibrating the second 
optical position information stored by the second optical position information storing means 
on the basis of a motion vector of the image information obtained by the imaging device in 
the second coordination system. The camera calibrating apparatus according to the present 

35 invention thus constructed as previously mentioned can calibrate the second optical position 
information by using the motion vector. 

In the camera calibrating apparatus according to the present invention, the 
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calibrating means includes: plane-projected image producing means for producing a 
plane-projected image from the image information obtained by the imaging device in the 
second coordination system; plane-projected image dividing means for dividing the 
plane-projected image produced by the plane-projected image producing means into a 
5 plurality of image segments; motion vector extracting means for extracting the motion 
vector from the image segments divided by the plane-projected image dividing means; 
calibration value calculating means for calculating a calibration value of the second optical 
position information stored by the second optical position information storing means on the 
basis of the motion vector extracted by the motion vector extracting means; and optical 

10 position information calibrating means for calibrating the second optical position 
information stored by the second optical position information storing means on the basis of 
the calibration value calculated by the calibration value calculating means. The camera 
calibrating apparatus according to the present invention thus constructed as previously 
mentioned can easily extract the motion vector from the image segments. 

15 In the camera calibrating apparatus according to the present invention, a dividing 

marker located in the second coordination system in a predetermined relationship with the 
position of the housing unit represented by the second position information stored by the 
second position information storing means. The plane-projected image dividing means is 
adapted to divide the plane-projected image produced by the plane-projected image 

20 producing means into a plurality of image segments on the basis of the image information of 
the dividing marker obtained by the imaging device. The camera calibrating apparatus 
according to the present invention thus constructed as previously mentioned can accurately 
divide the plane-projected image by using the dividing marker. 

In accordance with the present invention, there is provided a camera calibrating 

25 apparatus to be operative in combination with an imaging device which includes a housing 
unit and an optical section supported by the housing unit to obtain image information 
thorough the optical section, and adapted to calibrate position information of the optical 
section, comprising: first housing position information storing means for storing first 
housing position information indicative of the position of the housing unit in the first 

30 coordination system in which a revising marker is located; second housing position 
information storing means for storing second housing position information indicative of the 
position of the housing unit in the second coordination system in which an automotive 
vehicle is located; first optical position information producing means for producing first 
optical position information indicative of the position of the optical section in the first 

35 coordination system on the basis of the image information of the revising marker obtained 
by the imaging device; first optical position information storing means for storing the first 
optical position information produced by the first optical position information producing 
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means; second optical position information producing means for producing second optical 
position information indicative of the position of the optical section in the second 
coordination system from the second housing position information stored by the second 
housing position information storing means on the basis of the first housing position 
5 information stored by the first housing position information storing means and the first 
optical position information stored by the first optical position information storing means; 
estimated location information producing means for producing estimated location 
information indicative of the position of the automotive vehicle to an image coordination 
system of the imaging device on the basis of the second optical position information 

10 produced by the second optical position information producing means; second optical 
position information storing means for storing the second optical position information 
produced by the second optical position information producing means; estimated location 
information storing means for storing the estimated location information produced by the 
estimated location information estimating means; and calibrating means for calibrating the 

15 second optical position information stored by the second optical position information storing 
means on the basis of the image information of the automotive vehicle obtained by the 
imaging device and the estimated location information stored by the estimated location 
information storing means. The camera calibrating apparatus according to the present 
invention thus constructed as previously mentioned can calibrate the second optical position 

20 information by using a portion of the automotive vehicle. 

In the camera calibrating apparatus according to the present invention, the 
calibrating means includes: image location information extracting means for extracting an 
image location information indicative of an image location of the automotive vehicle in the 
image coordination system of the imaging device on the basis of the image information of 

25 the automotive vehicle obtained by the imaging device; calibration value calculating means 
for calculating a calibration value of the second optical position information stored by the 
second optical position information storing means on the basis of the image location 
information extracted by the image location information extracting means and the estimated 
location information stored by the estimated location information storing means; and optical 

30 position information calibrating means for calibrating the second optical position 
information stored by the second optical position information storing means on the basis of 
the calibration value calculated by the calibration value calculating means. The camera 
calibrating apparatus according to the present invention thus constructed as previously 
mentioned can calculate the calibration value of the second optical position information by 

35 using a portion of the automotive vehicle. 

In the camera calibrating apparatus according to the present invention, the 
calibration value calculating means includes: superimposing means for superimposing a 
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profile line of the automotive vehicle represented by the image location information on a 
profile line of the automotive vehicle represented by the estimated location information; 
extracting means for extracting a plurality of points from the overlapped profile lines of the 
automotive vehicle superimposed by the superimposing means, and calculating means for 
5 calculating a calibration value of the second optical position information by comparing the 
points of the image location information with the points of the estimated location 
information. The camera calibrating apparatus according to the present invention thus 
constructed as previously mentioned can extract the points from the profile line of the 
automotive vehicle to ensure that the second optical position information is extracted. 

10 In the camera calibrating apparatus according to the present invention, the imaging 

device is mounted on an automotive vehicle. The camera calibrating apparatus according 
to the present invention thus constructed as previously mentioned can calibrate parameters 
of the optical section to accurately detect the position of an object located on the road even 
if the housing unit is mounted on the inaccurate position of the automotive vehicle. 

15 In accordance with the present invention, there is provided an imaging system 

comprises a camera calibrating apparatus. The imaging system according to the present 
invention thus constructed as previously mentioned can calibrate parameters of the optical 
section mounted on an automotive vehicle or the like. 

In accordance with the present invention, there is provided an imaging control 

20 system comprises a camera calibrating apparatus. The imaging control system according 
to the present invention thus constructed as previously mentioned can calibrate parameters 
of the optical section mounted on an automotive vehicle or the like. 

In accordance with the present invention, there is provided a camera calibrating 
method of calibrating the position of an optical section supported by a housing unit of a 

25 camera for obtaining image information through the optical section, comprising: a first 
housing position information storing step of storing first housing position information 
indicative of the position of the housing unit in the first coordination system; a second 
housing position information storing step of storing second housing position information 
indicative of the position of the housing unit in the second coordination system; a first 

30 optical position information producing step of producing first optical position information 
indicative of the position of the optical section in the first coordination system on the basis 
of the image information obtained by the imaging device in the first coordination system; a 
first optical position information storing step of storing the first optical position information 
produced in the first optical position information producing step; a second optical position 

35 information producing step of producing second optical position information indicative of 
the position of the optical section in the second coordination system from the second 
housing position information stored in the second housing position information storing step 
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on the basis of the first housing position information stored in the first housing position 
information storing step and the first optical position information stored in the first optical 
position information storing step; a second optical position information storing step of 
storing the second optical position information produced in the second optical position 
5 information producing step; and a calibrating step of calibrating the second optical position 
information stored in the second optical position information storing step on the basis of the 
image information obtained by the imaging device in the second coordination system. The 
camera calibrating method according to the present invention thus constructed as previously 
mentioned can calibrate parameters of the optical section mounted on an automotive vehicle 

10 or the like to accurately detect the position of an object located on the road. 

In accordance with the present invention, there is provided a camera calibrating 
method of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, comprising: an optical 
position information storing step of storing optical position information indicative of the 

15 position of the optical section to a predetermined coordinate system; and a calibrating step 
of calibrating the optical position information stored in the optical position information 
storing step on the basis of the image information obtained by the imaging device on the 
predetermined coordinate system. The camera calibrating method according to the present 
invention thus constructed as previously mentioned can calibrate parameters of the optical 

20 section mounted on an automotive vehicle or the like to accurately detect the position of an 
object located on the road. 

In accordance with the present invention, there is provided a camera calibrating 
method of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, comprising: a first 

25 housing position information storing step of storing first housing position information 
indicative of the position of the housing unit in the first coordination system in which a 
revising marker is located; a second housing position information storing step of storing 
second housing position information indicative of the position of the housing unit in the 
second coordination system in which a calibrating marker is located; a first optical position 

30 information producing step of producing first optical position information indicative of the 
position of the optical section in the first coordination system on the basis of the image 
information of the revising marker obtained by the imaging device; a first optical position 
information storing step of storing the first optical position information produced in the first 
optical position information producing step; a second optical position information producing 

35 step of producing second optical position information indicative of the position of the 
optical section in the second coordination system from the second housing position 
information stored in the second housing position information storing step on the basis of 
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the first housing position information stored in the first housing position information storing 
step and the first optical position information stored in the first optical position information 
storing step; an estimated location information producing step of producing estimated 
location information indicative of the position of the calibrating marker to an image 
5 coordination system of the imaging device on the basis of the second optical position 
information produced in the second optical position information producing step; a second 
optical position information storing step of storing the second optical position information 
produced in the second optical position information producing step; an estimated location 
information storing step of storing the estimated location information produced in the 

10 estimated location information estimating step; and a calibrating step of calibrating the 
second optical position information stored in the second optical position information storing 
step on the basis of the image information of the calibrating marker obtained by the imaging 
device and the estimated location information stored in the estimated location information 
storing step. The camera calibrating method according to the present invention thus 

15 constructed as previously mentioned can calibrate the second optical position information 
by using the simple calibrating marker. 

In accordance with the present invention, there is provided a camera calibrating 
method of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, comprising: a first 

20 housing position information storing step of storing first housing position information 
indicative of the position of the housing unit in the first coordination system in which a 
revising marker is located; a second housing position information storing step of storing 
second housing position information indicative of the position of the housing unit in the 
second coordination system; a first optical position information producing step of producing 

25 first optical position information indicative of the position of the optical section in the first 
coordination system on the basis of the image information obtained by the imaging device 
and indicative of the revising marker; a first optical position information storing step of 
storing the first optical position information produced in the first optical position 
information producing step; a second optical position information producing step of 

30 producing second optical position information indicative of the position of the optical 
section in the second coordination system from the second housing position information 
stored in the second housing position information storing step on the basis of the first 
housing position information stored in the first housing position information storing step 
and the first optical position information stored in the first optical position information 

35 storing step; a second optical position information storing step of storing the second optical 
position information produced in the second optical position information producing step; 
and a calibrating step of calibrating the second optical position information stored in the 
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second optical position information storing step on the basis of a motion vector of the image 
information obtained by the imaging device in the second coordination system. The 
camera calibrating method according to the present invention thus constructed as previously 
mentioned can calibrate the second optical position information by using the motion vector 
5 In accordance with the present invention, there is provided a camera calibrating 

method of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, comprising: a first 
housing position information storing step of storing first housing position information 
indicative of the position of the housing unit in the first coordination system in which a 

10 revising marker is located; a second housing position information storing step of storing 
second housing position information indicative of the position of the housing unit in the 
second coordination system in which an automotive vehicle is located; a first optical 
position information producing step of producing first optical position information 
indicative of the position of the optical section in the first coordination system on the basis 

15 of the image information of the revising marker obtained by the imaging device; a first 
optical position information storing step of storing the first optical position information 
produced in the first optical position information producing step; a second optical position 
information producing step of producing second optical position information indicative of 
the position of the optical section in the second coordination system from the second 

20 housing position information stored in the second housing position information storing step 
on the basis of the first housing position information stored in the first housing position 
information storing step and the first optical position information stored in the first optical 
position information storing step; an estimated location information producing step of 
producing estimated location information indicative of the position of the automotive 

25 vehicle to an image coordination system of the imaging device on the basis of the second 
optical position information produced in the second optical position information producing 
step; a second optical position information storing step of storing the second optical position 
information produced in the second optical position information producing step; an 
estimated location information storing step of storing the estimated location information 

30 produced in the estimated location information estimating step; and a calibrating step of 
calibrating the second optical position information stored in the second optical position 
information storing step on the basis of the image information of the automotive vehicle 
obtained by the imaging device and the estimated location information stored in the 
estimated location information storing step. The camera calibrating method according to 

35 the present invention thus constructed as previously mentioned can calibrate the second 
optical position information by using a portion of the automotive vehicle. 

In the camera calibrating apparatus according to the present invention, the imaging 

12 



device is mounted on an automotive vehicle. The camera calibrating method according to 
the present invention thus constructed as previously mentioned can calibrate parameters of 
the optical section mounted on an automotive vehicle or the like to accurately detect the 
position of an object located on the road even if the housing unit is mounted on the 
5 inaccurate position of the automotive vehicle. 

In accordance with the present invention, there is provided a camera calibrating 
program of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, and allowing a 
computer to execute: a first housing position information storing step of storing first housing 

10 position information indicative of the position of the housing unit in the first coordination 
system; a second housing position information storing step of storing second housing 
position information indicative of the position of the housing unit in the second coordination 
system; a first optical position information producing step of producing first optical position 
information indicative of the position of the optical section in the first coordination system 

15 on the basis of the image information obtained by the imaging device in the first 
coordination system; a first optical position information storing step of storing the first 
optical position information produced in the first optical position information producing 
step; a second optical position information producing step of producing second optical 
position information indicative of the position of the optical section in the second 

20 coordination system from the second housing position information stored in the second 
housing position information storing step on the basis of the first housing position 
information stored in the first housing position information storing step and the first optical 
position information stored in the first optical position information storing step; a second 
optical position information storing step of storing the second optical position information 

25 produced in the second optical position information producing step; and a calibrating step of 
calibrating the second optical position information stored in the second optical position 
information storing step on the basis of the image information obtained by the imaging 
device in the second coordination system. The camera calibrating program according to 
the present invention thus constructed as previously mentioned can calibrate parameters of 

30 the optical section mounted on an automotive vehicle or the like to accurately detect the 
position of an object located on the road. 

In accordance with the present invention, there is provided a camera calibrating 
program of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, and allowing a 

35 computer to execute: an optical position information storing step of storing optical position 
information indicative of the position of the optical section to a predetermined coordinate 
system; and a calibrating step of calibrating the optical position information stored in the 
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optical position information storing step on the basis of the image information obtained by 
the imaging device on the predetermined coordinate system. The camera calibrating 
program according to the present invention thus constructed as previously mentioned can 
calibrate parameters of the optical section mounted on an automotive vehicle or the like to 
5 accurately detect the position of an object located on the road. 

In accordance with the present invention, there is provided a camera calibrating 
program of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, and allowing a 
computer to execute: a first housing position information storing step of storing first housing 

10 position information indicative of the position of the housing unit in the first coordination 
system in which a revising marker is located; a second housing position information storing 
step of storing second housing position information indicative of the position of the housing 
unit in the second coordination system in which a calibrating marker is located; a first 
optical position information producing step of producing first optical position information 

15 indicative of the position of the optical section in the first coordination system on the basis 
of the image information of the revising marker obtained by the imaging device; a first 
optical position information storing step of storing the first optical position information 
produced in the first optical position information producing step; a second optical position 
information producing step of producing second optical position information indicative of 

20 the position of the optical section in the second coordination system from the second 
housing position information stored in the second housing position information storing step 
on the basis of the first housing position information stored in the first housing position 
information storing step and the first optical position information stored in the first optical 
position information storing step; an estimated location information producing step of 

25 producing estimated location information indicative of the position of the calibrating marker 
to an image coordination system of the imaging device on the basis of the second optical 
position information produced in the second optical position information producing step; a 
second optical position information storing step of storing the second optical position 
information produced in the second optical position information producing step; an 

30 estimated location information storing step of storing the estimated location information 
produced in the estimated location information estimating step; and a calibrating step of 
calibrating the second optical position information stored in the second optical position 
information storing step on the basis of the image information of the calibrating marker 
obtained by the imaging device and the estimated location information stored in the 

35 estimated location information storing step. The camera calibrating program according to 
the present invention thus constructed as previously mentioned can calibrate the second 
optical position information by using the simple calibrating marker. 
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In accordance with the present invention, there is provided a camera calibrating 
program of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, and allowing a 
computer to execute: a first housing position information storing step of storing first housing 
5 position information indicative of the position of the housing unit in the first coordination 
system in which a revising marker is located; a second housing position information storing 
step of storing second housing position information indicative of the position of the housing 
unit in the second coordination system; a first optical position information producing step of 
producing first optical position information indicative of the position of the optical section 

10 in the first coordination system on the basis of the image information obtained by the 
imaging device and indicative of the revising marker; a first optical position information 
storing step of storing the first optical position information produced in the first optical 
position information producing step; a second optical position information producing step of 
producing second optical position information indicative of the position of the optical 

15 section in the second coordination system from the second housing position information 
stored in the second housing position information storing step on the basis of the first 
housing position information stored in the first housing position information storing step 
and the first optical position information stored in the first optical position information 
storing step; a second optical position information storing step of storing the second optical 

20 position information produced in the second optical position information producing step; 
and a calibrating step of calibrating the second optical position information stored in the 
second optical position information storing step on the basis of a motion vector of the image 
information obtained by the imaging device in the second coordination system. The 
camera calibrating program according to the present invention thus constructed as 

25 previously mentioned can calibrate the second optical position information by using the 
motion vector. 

In accordance with the present invention, there is provided a camera calibrating 
program of calibrating the position of an optical section supported by a housing unit of a 
camera for obtaining image information through the optical section, and allowing a 

30 computer to execute: a first housing position information storing step of storing first housing 
position information indicative of the position of the housing unit in the first coordination 
system in which a revising marker is located; a second housing position information storing 
step of storing second housing position information indicative of the position of the housing 
unit in the second coordination system in which an automotive vehicle is located; a first 

35 optical position information producing step of producing first optical position information 
indicative of the position of the optical section in the first coordination system on the basis 
of the image information of the revising marker obtained by the imaging device; a first 

15 



optical position information storing step of storing the first optical position information 
produced in the first optical position information producing step; a second optical position 
information producing step of producing second optical position information indicative of 
the position of the optical section in the second coordination system from the second 
5 housing position information stored in the second housing position information storing step 
on the basis of the first housing position information stored in the first housing position 
information storing step and the first optical position information stored in the first optical 
position information storing step; an estimated location information producing step of 
producing estimated location information indicative of the position of the automotive 

10 vehicle to an image coordination system of the imaging device on the basis of the second 
optical position information produced in the second optical position information producing 
step; a second optical position information storing step of storing the second optical position 
information produced in the second optical position information producing step; an 
estimated location information storing step of storing the estimated location information 

15 produced in the estimated location information estimating step; and a calibrating step of 
calibrating the second optical position information stored in the second optical position 
information storing step on the optical section basis of the image information of the 
automotive vehicle obtained by the imaging device and the estimated location information 
stored in the estimated location information storing step. The camera calibrating program 

20 according to the present invention thus constructed as previously mentioned can calibrate 
the second optical position information by using a portion of the automotive vehicle. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The features and advantages of a camera calibrating apparatus according to the 
25 present invention will be more clearly understood from the following description taken in 
conjunction with the accompanying drawings: 

FIG 1 is a block diagram showing a camera calibrating apparatus according to the 
first embodiment of the present invention and a camera exemplified as an imaging device; 

FIG 2 is a perspective view showing the first coordination system in which the 
30 camera shown in FIG 1 is located; 

FIG 3 is a perspective view showing the second coordination system in which the 
camera shown in FIG 1 is located; 

FIG 4 is a perspective view showing a coordination system of the camera shown in 

FIG 1; 

35 FIG 5 is a plan view showing an image coordination system of the camera shown 

in FIG 1; 

FIG 6 is a perspective view showing a parallel translation of the camera shown in 
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FIG 1; 

FIG 7 is a perspective view showing a rotational motion of the camera shown in 

FIG 1; 

FIG 8 is a side view showing a parallel translation of the camera shown in FIG 1; 
5 FIG 9 is a side view showing a rotational motion of the camera shown in FIG 1 ; 

FIG 10 is a block diagram showing an image location information extracting 
section of the camera calibrating apparatus shown in FIG 1; 

FIG 11 is a block diagram showing a computer and ECU to ensure the camera 
calibrating apparatus shown in FIG 1 ; 
10 FIG 12 is a block diagram showing a camera unit of the camera calibrating 

apparatus shown in FIG 1; 

FIG 13 is a flow chart showing a calibrating operation of the camera calibrating 
apparatus shown in FIG 1 ; 

FIG 14 is a block diagram showing a camera calibrating apparatus according to the 
15 second embodiment of the present invention and a camera exemplified as an imaging 
device; 

FIG 15 a plan view showing an image coordination system of the camera shown in 

FIG 14; 

FIG 16 is a flow chart showing a calibrating operation of the camera calibrating 
20 apparatus shown in FIG 14; 

FIG 17 is a block diagram showing the camera calibrating apparatus according to 
the third embodiment of the present invention and a camera exemplified as an imaging 
device; 

FIG 18 is a perspective view showing the first coordination system in which the 
25 camera shown in FIG 17 is located; 

FIG 19 is a perspective view showing the second coordination system in which the 
camera shown in FIG 17 is located; 

FIG 20 is a perspective view showing a coordination system of the camera shown 
in FIG 17; 

30 FIG 21 is a side view showing an imaginary camera of the camera calibrating 

apparatus shown in FIG 17; 

FIG 22 is a schematic view showing an operation of the calibrating section of the 
camera calibrating apparatus shown in FIG 17; 

FIG 23 is a schematic view showing an operation of the calibrating section of the 
35 camera calibrating apparatus shown in FIG 17; 

FIG 24 is a schematic view showing an operation of the calibrating section of the 
camera calibrating apparatus shown in FIG 17; 
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FIG 25 is a schematic view showing an operation of the calibrating section of the 
camera calibrating apparatus shown in FIG 17; 

FIG 26 is a block diagram showing a computer and ECU to ensure the camera 
calibrating apparatus shown in FIG 17; 
5 FIG 27 is a block diagram showing a camera unit of the camera calibrating 

apparatus shown in FIG 17; 

FIG 28 is a flow chart showing a calibrating operation of the camera calibrating 
apparatus shown in FIG 17; 

FIG 29 is a block diagram showing a camera calibrating apparatus according to the 
10 fourth embodiment of the present invention and a camera exemplified as an image device; 

FIG 30 is a perspective view showing the first coordination system in which the 
camera shown in FIG 29 is located; 

FIG 31 is a perspective view showing the second coordination system in which the 
camera shown in FIG 29 is located; 
15 FIG 32 is a perspective view showing a coordination system of the camera shown 

in FIG 29; 

FIG 33 is a plan view showing an image coordination system of the camera shown 
in FIG 29; 

FIG 34 is a schematic view showing an operation of the calibrating section of the 
20 camera calibrating apparatus shown in FIG 29; 

FIG 35 is a schematic view showing an operation of the calibrating section of the 
camera calibrating apparatus shown in FIG 29; 

FIG 36 is a block diagram showing a computer and ECU to ensure the camera 
calibrating apparatus shown in FIG 29; 
25 FIG 37 is a block diagram showing a camera unit of the camera calibrating 

apparatus shown in FIG 29; 

FIG 38 is a flow chart showing a calibrating operation of the camera calibrating 
apparatus shown in FIG 29; 

FIG 39 is a block diagram of a conventional camera calibrating apparatus and a 
30 camera exemplified as an imaging device; 

FIG 40 is a perspective view showing the first coordination system in which the 
camera shown in FIG 39 is located; and 

FIG 41 is a perspective view showing the second coordination system in which the 
camera shown in FIG 39 is located. 

35 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The embodiments of the camera calibrating apparatus according to the present 
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invention will now be described in detail in accordance with accompanying drawings, 
(first embodiment) 

Referring now to the drawings, in particular to FIGS. 1 to 13, there is shown a first 
embodiment of a camera calibrating apparatus according to the present invention. 
5 The construction of the first embodiment of the camera calibrating apparatus 

according to the present invention will now be described hereinafter. As shown in FIGS. 1 
to 3, the camera calibrating apparatus 100 is electrically connected to a camera 110 
exemplified as an imaging device. The camera 110 includes a housing unit 111 and an 
optical section 112 supported by the housing unit 111 to obtain image information through 

10 the optical section 112. 

The camera calibrating apparatus 100 comprises a first housing position 
information storing section 115 for storing first housing position information indicative of 
the position of the housing unit 111 in the first coordination system 101, a second housing 
position information storing section 116 for storing second housing position information 

15 indicative of the position of the housing unit 111 in the second coordination system 102, a 
revising marker position information storing section 125 for storing revising marker position 
information indicative of the position of a revising marker 105 in the first coordination 
system 101, and a calibrating marker position information storing section 126 for storing 
calibrating marker position information indicative of the position of a calibrating marker 

20 106 in the second coordination system 102. 

The first coordination system 101 is defined in a first working space such as a 
camera production plant to have Xi-axis, Yi-axis, and Z r axis, while the revising marker 
105 laid out in the first working space causes the camera calibrating apparatus 100 to revise 
the camera 110 located in the first working space. The revising marker 105 is constituted 

25 by a plurality of marking objects which are respectively located in predetermined positions 
on the first coordination system 101, and which are within a viewing field of the camera 110 
located in the first working space. 

The second coordination system 102 is defined in a second working space such as 
an automotive vehicle production plant to have X 2 -axis, Y 2 -axis, and Z 2 -axis, while X 2 -axis 

30 and Y 2 -axis collectively define X 2 -Y 2 plane which is represented by a road surface 102a on 
which an automotive vehicle 108 is located. The calibrating marker 106 is constituted by 
two or more marking objects which are respectively located at predetermined positions of 
the road surface 102a in the second coordination system 102. 

The camera calibrating apparatus 100 is adapted to revise the camera 110 in the 

35 first working space. The camera 110 is located at a predetermined position on the first 
coordination system 101, while the first housing position information storing section 115 is 
adapted to store first housing position information indicative of the position of the housing 
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unit 111. Here, the term "the calibration of the camera 110" is intended to indicate the 
calculation of the accurate position of the optical section 112 of the camera 110 mounted on 
the automotive vehicle 108. 

The camera 110 revised by the camera calibrating apparatus 110 is mounted on the 
5 automotive vehicle 108 to be located at a predetermined position on the second coordination 
system 102 in the second working space, while the second housing position information 
storing section 116 is adapted to store second housing position information indicative of the 
position of the housing unit 111. Here, the second housing position information indicates 
an accurate position in which the housing unit 111 of the camera 110 is located when the 

10 camera 110 is mounted on the automotive vehicle 108. 

The camera calibrating apparatus 100 further comprises a first optical position 
information producing section 117 for producing first optical position information indicative 
of the position of the optical section 112 in the first coordination system 101, and a first 
optical position information storing section 118 for storing the first optical position 

15 information produced by the first optical position information producing section 117. 

The first optical position information producing section 117 is adapted to calculate 
the position of the optical section 112 in the first coordination system 101 from the 
calibrating marker position information stored in the calibrating marker position information 
storing section 125 on the basis of the image information of the calibrating marker 105 

20 obtained by the camera 110. Here, the term "the position of the optical section 112" is 
intended to indicate an optical center of the optical section 112 and the position of an optical 
axis, and other optical parameters. The calculation of the position of the optical section 
112 in the first coordination system 101 is performed on the basis of a method disclosed in 
the document 1 (R. Tsai, A versatile camera calibration technique for high-accuracy 3D 

25 machine vision metrology using off-the-shelf TV cameras and lenses, IEEE Journal of 
Robotics and Automation, RA-3 (4): 323-344, 1987). 

The camera calibrating apparatus 100 further comprises a second optical position 
information producing section 120 for producing second optical information indicative of 
the position of the optical section 112 in the second coordination system 102, and a second 

30 optical position information storing section 130 for storing the second optical position 
information produced by the second optical position information producing section 120. 

The second optical position information producing section 120 is adapted to 
produce second optical position information indicative of the position of the optical section 
112 in the second coordination system 102 from the second housing position information 

35 stored in the second housing position information storing section 116 on the basis of the first 
housing position information stored by the first housing position information storing section 
115 and the first optical position information stored in the first optical position information 
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storing section 118. 

The second optical position information producing section 120 is adapted to 
calculate the position of the optical section 112 in the second coordination system 102 on 
the basis of a following method. 
5 The method is of comparing the position of the housing unit 111 to the first 

coordination system 101 with the position of the optical section 112 in the first coordination 
system 101 to obtain positional information between the housing unit 111 and the optical 
section 112, and to calculate information indicative of the position of the optical section 112 
in the second coordination system 102 from the position of the housing unit 111 in the 

10 second coordination system 102 on the basis of the positional information between the 
housing unit 111 and the optical section 112. Consequently, the second optical position 
information indicates the position of the optical section 112 under the condition that the 
housing unit 111 is located at an accurate position of the automotive vehicle 108. 

As shown in FIG 4, the camera 110 mounted on the automotive vehicle 108 in the 

15 second working space defines a camera coordination system 113 having x-axis, y-axis, and 
z-axis on the basis of the second optical position information. The origin point of the 
camera coordination system 113 is in register with the optical center of the optical section 
112. The x-axis of the camera coordination system 113 extends in a horizontal direction of 
the camera 110. The y-axis of the camera coordination system 113 extends in a vertical 

20 direction of the camera 110. The z-axis of the camera coordination system 113 is in axial 
alignment with the optical axis of the optical section 112. 

The image coordination system 114 is defined on a plane spaced apart from the 
origin point of the camera coordination system 113 with a focal length "f ' in the direction of 
the z-axis of the camera coordination system 113. The image coordination system 114 has 

25 p-axis and q-axis. The camera 110 is adapted to obtain, as image information, an optical 
image focused on the image coordination system 114 through the optical section 112. 

The camera calibrating apparatus 100 further comprises an estimated location 
information producing section 140 for producing estimated location information of the 
calibrating marker 106 to the image coordination system 114 of the camera 110, and an 

30 estimated location information storing section 150 for storing the estimated location 
information produced by the estimated location information producing section 140. 

The estimated location information producing section 140 is adapted to produce 
estimated location information of the calibrating marker 106 to the image coordination 
system 114 of the camera 110 from the position information of the calibrating marker 106 

35 stored in the calibrating marker position information storing section 126 on the basis of the 
second optical position information produced by the second optical position information 
producing section 120. The calculation of the estimated location of the calibrating marker 
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106 to the image coordination system 114 of the camera 110 is performed on the basis of the 
above mentioned document 1. 

As shown in FIG. 5, the calibrating marker 106 located in the second coordination 
system 102 is focused on the image location Pn' (n = 1, 2, 3, 4, 5, 6) of the image 
5 coordination system 114 through the optical section 112. Here, the image location Pn* is in 
register with the estimated image location Pn (n = 1, 2, 3, 4, 5, 6) calculated by the 
estimated image location calculating section 140 under the condition that the housing unit 
111 is located at an accurate position of the automotive vehicle 108, more specifically, the 
housing unit 111 is located at the position represented by the second housing position 

10 information with no deviation to the accurate position. However, the housing unit 111, in 
general, is located at an inaccurate position with deviation to the accurate position. This 
leads to the fact that the second optical position information is produced with deviation to 
the accurate position. As a result, the estimated location Pn is spaced apart from the image 
location Pn f of the image coordination system 114. 

15 The second optical position information includes six parameters indicative of a 

parallel displacement and a rotational motion of the camera coordination system 1 1 3 to the 
second coordination system 102. The six parameters are constituted by the parallel 
displacement components of the X 2 -axis, the Y 2 -axis, and the Z 2 -axis shown in FIG 6 and 
the rotational motion of the x-axis, the y-axis, and the z-axis shown in FIG 7. The 

20 deviation of the second optical position information includes components of the parallel 
displacement and rotational motion. The housing unit 111 is, in general, located on the 
automotive vehicle with the parallel displacement which is within the range of a few 
centimeters, and the rotational motion which is within the range of a few degrees. 

The following description will be directed to the case that the image information of 

25 the road surface 102a obtained by the camera 110 is displayed in conjunction with an 
additional line when the automotive vehicle 108 is operated by an operator. Here, the 
housing unit 111 is located at the position of the automotive vehicle 108 which is spaced 
apart from the road surface 302a, while the additional line is located at the position spaced 
apart from a rear portion of the automotive vehicle 108. The height of the housing unit 111 

30 to the road surface 302a is equal to 1,000 millimeters, while the distance of the additional 
line to the rear portion is equal to 3000 millimeters. 

When the housing unit 111 is located at the position spaced apart from a design 
position of the automotive vehicle 108 in the Y-axis direction, more specifically, the parallel 
shift displacement of the housing unit 111 to the design position is equal to 50 millimeters 

35 as shown in FIG 8, the displacement of the additional line is equal to 50 millimeters in 
Y 2 -axis direction. In this case, the elongation value of the image lpcation Pn f to the 
estimated location Pn on the image coordination system 114 shown in FIG 5 can be 
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negligible. 

When, on the other hand, the housing unit 111 is mounted on the automotive 
vehicle 108 with a rotational displacement of the housing unit 111 to the X 2 -axis direction, 
more specifically, the rotational displacement of the housing unit 111 to the X 2 -axis 
5 direction is equal to one degree as shown in FIG 9, the additional line is located with a 
displacement of 184 millimeters in the Y 2 -axis direction. In this case, the elongation value 
of the image location Pn T to the estimated location Pn on the image coordination system 114 
shown in FIG 5 cannot be negligible. 

As will be seen from the above description, the second optical position information 
10 may be calibrated in consideration with only the rotational displacement of the optical 
section 112. 

The camera calibrating apparatus 100 further comprises a calibrating section 160 
for calibrating the second optical position information stored in the second optical position 
information storing section 130 to order to calibrate the above mentioned deviation of the 
15 second optical position information. 

The calibrating section 160 is adapted to calibrate the second optical position 
information stored in the second optical position information storing section 130 on the 
basis of the image information of the calibrating marker 106 obtained by the camera 110 
and the estimated location information stored in the estimated location information storing 
20 section 150. 

The calibrating section 160 includes an image location information extracting 
section 170 for extracting image location information indicative of an image location of the 
calibrating marker 106 to the image coordination system 114 of the camera 110 on the basis 
of the image information of the calibrating marker 106 obtained by the camera 110, a 

25 calibration value calculating section 180 for calculating a calibration value of the second 
optical position information stored in the second optical position information storing section 
130 on the basis of the image location information extracted by the image location 
information extracting section 170 and the estimated location information stored in the 
estimated location information storing section 150, and an optical position information 

30 calibrating section 190 for calibrating the second optical position information stored in the 
second optical position information storing section 130 on the basis of the calibration value 
calculated by the calibration value calculating section 180. 

The image location information extracting section 170 includes an image 
displaying section 171 for displaying an image of the calibrating marker 106 obtained by the 

35 camera 110, and an image location specifying section 172 for specifying the image location 
Pn' of the calibrating marker 106 in the image of the calibrating marker 106 displayed by the 
image displaying means 171 in order to extract the image location information from the 
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image information. 

As shown in FIG 10, the image displaying section 171 is adapted to display both 
the image of the calibrating marker 106 obtained by the camera 110 and a cursor 174 for 
specifying the image location Pn f of the calibrating marker 106. 
5 The image location specifying section 172 includes an up-arrow key 175a, a 

down-arrow key 175b, a left-arrow key 175c, and a right-arrow key 175d, all of which are 
capable of allowing the cursor 174 displayed on the image information displaying section 
171 to be moved in the respective directions, an enter key 176 for deciding the position of 
the cursor 174 as the image location Pn' of the calibrating marker 106, and a number 

10 displaying section 177 for displaying number "n" of the marking object of the calibrating 
marker 106, and an increment key 178a and a decrement key 178b each for changing the 
number "n" displayed on the number displaying section 177. 

The calibration value calculating section 180 is adapted to calculate the calibration 
value of the second optical position information on the basis of the following method. 

15 The following direction will be directed to the case that the deviation of the 

rotational component of the second optical position information is considered as an overall 
deviation of the second optical position information. 



N 



(pn (Rnpn + R23g„ + R33f)-f(Rl \p + R2\q + Ril f)f 

+ [qn(Rnp + R23g„ + Rnf)-f(R\ip + Rnq + R32f)f _ 



(1) 
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(p n \ q n ') and (p n , q n ) are linked by the following formula (2) on the basis of the 
formula (1). 



P=f 



Rnp + R2\q + Riif 
Rnp + R23q + R33f 

,_ f Rup + R22q + R32f 

q Rnp + R23q + Ri3f 



(2) 
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R n toR 3 3and 9 , <f> , <p are linked by the following formulas (3), (4), and (5). 
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Rx = 



Ry = 



Rz = 



10 0 
0 cos 0 sin 9 
0 - sin 6 cos 6 
cos <j> 0 - sin ^ 

0 1 0 
sin <f> 0 cos <j> 
cos <p sin <p 0 
— sin ^ cos ^ 0 

0 0 1 



(3) 



R = RxxRyxRz 



(4) 



i?ll i?12 Rl3 
R= R2\ R22 R23 
R3\ R32 R33 



(5) 



In this embodiment, the calibrating marker 106 is constituted by six marking 
objects, however, the calibrating marker 106 may be constituted by two or more marking 
objects. On the other hand, the calibration value calculating section 180 can calculate 8 , 
10 $ , and <p if the calibrating marker 106 is constituted by two marking objects. 
Additionally, the calibration value calculating section 180 can calculate 9 , , and </? at 
a relatively high accuracy if the calibrating marker 106 is constituted by three or more 
marking objects. 

As shown in FIG 11, the camera calibrating apparatus 100 thus constructed as 
15 previously mentioned can be realized by comprising the computer 191 for calibrating the 
camera 110, and the ECU 192 constituted as an image controlling apparatus for controlling 
the camera 110. 

The computer 191 is shown in FIG 11(a) as including CPU, RAM, ROM, an 
input/output interface, and other elements. The computer 191 is electrically connected to 

20 the camera 110 at the first working space. In this embodiment, the computer 191 
constitutes each of the first housing position information storing section 115, the second 
housing position information storing section 116, the first optical position information 
producing section 117, the first optical position information storing section 118, the second 
optical position information producing section 120, the calibrating marker position 

25 information storing section 125, the calibrating marker position information storing section 
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126, the second optical position storing section 130, the estimated location information 
producing section 140, and the estimated location information storing section 150. 

The ECU 192 is shown in FIG 11(c) as including CPU, RAM, ROM, an 
input/output interface, and other elements. The ECU 192 is mounted on the automotive 
5 vehicle 108 to be electrically connected to the camera 110 in the second working space. In 
this embodiment, the ECU 192 constitutes each of the second optical position information 
storing section 130, the estimated location information storing section 150, and the 
calibrating section 160. 

As will be seen from FIG 11(b), the camera 110 is transferred to the second 

10 working space from the first working space with a recordable medium 193 such as for 
example CD-ROM and a magnetic disk. The recordable medium 193 has stored therein 
each of the second optical position information and the estimated location information to 
ensure that the each of the second optical position information and the estimated location 
information is effectively transferred to the ECU 192 from the computer 191. 

15 In this embodiment, the camera 110 and the recordable medium 193 are transferred 

to the second working space from the first working space, however, the camera 110, the 
second optical position information storing section 130, the estimated location information 
storing section 150, and the calibrating section 160 may be collectively transferred as a 
camera assembly 194 to the second working space from the first working space as will be 

20 seen from FIG 12. 

The following description will be directed to the operation of the first embodiment 
of the camera calibrating apparatus according to the present invention. The second optical 
position information is calibrated by the camera calibrating apparatus 100 through the steps 
shown in the FIG 13. 

25 The camera 110 is firstly located at the predetermined position on the first 

coordination system 101 in the first working space (in the step S101). The first housing 
position information, the second housing position information, the revising marker position 
information, and the calibrating marker position information are then stored in the first 
housing position information storing section 115, the second housing position information 

30 storing section 116, the revising marker position information storing section 125, and the 
calibrating marker position information storing section 126 (in the step S102), respectively. 
Here, the first housing position information, the second housing position information, the 
revising marker position information, and the calibrating marker position information are 
obtained on the basis of positions predetermined in the design process and measured by a 

35 measuring instrument. 

The image information of the revising marker 105 is then obtained by the camera 
110 (in the step S103). The first optical position information is then produced by the first 
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optical position information producing section 117 from the revising marker position 
information stored in the revising marker position information storing section 125 on the 
basis of the image information of the revising marker 105 obtained by the camera 110 (in 
the step S104). The first optical position information produced by the first optical position 
5 information producing section 117 is then stored in the first optical position information 
storing section 118 (in the step S105). 

The second optical position information is then produced by the second optical 
position information producing section 120 from the second housing position information 
stored in the second housing position information storing section 116 on the basis of the first 

10 housing position information stored in the first housing position information storing section 
115 and the first optical position information stored in the first optical position information 
storing section 118 (in the step S106). The second optical position information produced 
by the second optical position information producing section 120 is then stored in the 
second optical position information storing section 130 (in the step S107). 

15 The estimated location information indicative of the estimated location of the 

calibrating marker 106 is then produced by the estimated location information producing 
section 140 from the calibrating marker position information stored in the calibrating 
marker position information storing section 126 on the basis of the second optical position 
information produced by the second optical position information producing section 120 (in 

20 the step S108). The estimated location information produced by the estimated location 
information producing section 140 is stored in the estimated location information storing 
section 150 (in the step S109). 

The camera 110 and the recordable medium 193 are then transferred to the second 
working space from the first working space. The camera 110 is then mounted to the 

25 automotive vehicle 108 to be located at the predetermined position on the second 
coordination system 102 (in the step SI 10). 

The image information of the calibrating marker 106 is then obtained by the camera 
110 (in the step Sill). The image information of the calibrating marker 106 obtained by 
the camera 110 is then displayed on the image displaying section 171 (in the step SI 12) as 

30 shown in FIG 10. The image location Pn' of the calibrating marker is specified by the 
image location specifying section 172 (in the step SI 13). Here, the increment key 178a 
and the decrement key 178b are operated by an operator, while the number "n" of the 
marking object of the calibrating marker 106 displayed by the number displaying section 
177 in response to the increment key 178a and the decrement key 178b. The up-arrow key 

35 175a, the down-arrow key 175b, the left-arrow key 175c, and the right-arrow key 175d are 
operated by the operator, while the cursor 174 displayed on the image displaying section 
171 is moved in response to each of the up-arrow key 175a, the down-arrow key 175b, the 
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left-arrow key 175c, and the right-arrow key 175d. The enter key 176 is operated by the 
operator, while the position of the cursor 176 is decided as the image location Pn' of the 
number "n" of the calibrating marker 106 displayed on the number displaying section 177. 

The calibration value of the second optical position information stored in the 
5 second optical position information storing section 130 is then calculated by the calibration 
value calculating section 180 on the basis of the image location information extracted by the 
image location information extracting section 170 and the estimated location information 
stored in the estimated location information storing section 150 (in the step SI 14). 

The second optical position information stored in the second optical position 
10 information storing section 130 is finally calibrated by the optical position information 
calibrating section 190 on the basis of the calibration value calculated by the calibration 
value calculating section 180 (in the step SI 15). In this embodiment, the above mentioned 
steps 101 to 115 are executed as a program by a computer. 

From the above detail description, it will be understood that the present 
15 embodiment of the camera calibrating apparatus 100 according to the present invention can 
calibrate the optical parameters of the camera 110 mounted on the automotive vehicle 108 to 
ensure an accurate position of an object located on the road is detected. 

In this embodiment, the camera calibrating apparatus 100 according to the present 
invention can calibrate the second optical position by using the simple calibrating marker 
20 106. 

In this embodiment, the camera calibrating apparatus 100 according to the present 
invention can specify the image location of the calibrating marker 106 to ensure that the 
image location information indicative of the image location of the calibrating marker 106 is 
extracted. 
25 (second embodiment) 

Referring now to the drawings, in particular to FIGS. 14 to 16, there is shown a 
second embodiment of a camera calibrating apparatus according to the present invention. 

The construction of the second embodiment of the camera calibrating apparatus 
according to the present invention will now be described hereinafter. The constitutional 
30 elements of the second embodiment of the camera calibrating apparatus the same as those of 
the first embodiment of the camera calibrating apparatus will not be described but bear the 
same reference numerals and legends as those of the first embodiment of the camera 
calibrating apparatus. 

In FIG 14, the camera calibrating apparatus 200 comprises a calibrating section 
35 260 for calibrating the second optical position information stored in the second optical 
position information storing section 130. 

The calibrating section 260 is adapted to calibrate the second optical position 
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information stored in the second optical position information storing section 130 on the 
basis of the image information of the calibrating marker 106 obtained by the camera 110 
and the estimated location information stored in the estimated location information storing 
section 150. 

5 The calibrating section 260 includes an image location information extracting 

section 270 for extracting image location information indicative of an image location of the 
calibrating marker 106 to the image coordination system 114 of the camera 110 on the basis 
of the image information of the calibrating marker 106 obtained by the camera 110, a 
calibration value calculating section 280 for calculating a calibration value to the second 

10 optical position information stored in the second optical position information storing section 
130 on the basis of the image location information extracted by the image location 
information extracting section 270 and the estimated location information stored in the 
estimated location information storing section 150, and an optical position information 
calibrating section 290 for calibrating the second optical position information stored in the 

15 second optical position information storing section 130 on the basis of the calibration value 
calculated by the calibration value calculating section 280. 

The image location information extracting section 270 includes an estimated area 
information storing section 271 for storing estimated area information (see FIG 15) 
indicative of an estimated area 274 of the calibrating marker 106 to the image coordination 

20 system 114 of the camera 110, and an image location searching section 272 for searching the 
image location Pn' of the calibrating marker 106 from the image information of the 
calibrating marker 106 obtained by the camera 110 on the basis of the estimated area 
information stored by the estimated area information storing section 271 and the estimated 
location information stored by the estimated location information storing section 150 to 

25 ensure that the image location information is extracted. 

The calibration value calculating section 280 is adapted to calculate the calibration 
value to the second optical position information on the basis of a method which is the same 
as the method of the calibration value calculating section 180 of the first embodiment. 

The following description will be directed to the operation of the second 

30 embodiment of the camera calibrating apparatus according to the present invention. The 
second optical position information is calibrated by the camera calibrating apparatus 200 
through the following steps shown in FIG 16. 

The camera 110 is firstly located at the predetermined position on the first 
coordination system 101 in a first working space (in the step S201). The first housing 

35 position information, the second housing position information, the revising marker position 
information, and the calibrating marker position information are then stored in the first 
housing position information storing section 115, the second housing position information 
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storing section 116, the revising marker position information storing section 125, and the 
calibrating marker position information storing section 126 (in the step S202), respectively. 
Here, the first housing position information, the second housing position information, the 
revising marker position information, and the calibrating marker position information are 
5 obtained on the basis of positions predetermined in the design process and measured by a 
measuring instrument. 

The image information of the revising marker 105 is then obtained by the camera 
110 (in the step S203). The first optical position information is then produced by the first 
optical position information producing section 117 from the revising marker position 

10 information stored in the revising marker position information storing section 125 on the 
basis of the image information of the revising marker 105 obtained by the camera 110 (in 
the step S204). The first optical position information produced by the first optical position 
information producing section 117 is then stored in the first optical position information 
storing section 118 (in the step S205). 

15 The second optical position information is then produced by the second optical 

position information producing section 120 from the second housing position information 
stored in the second housing position information storing section 116 on the basis of the first 
housing position information stored in the first housing position information storing section 
115 and the first optical position information stored in the first optical position information 

20 storing section 118 (in the step S206). The second optical position information produced 
by the second optical position information producing section 120 is then stored in the 
second optical position information storing section 130 (in the step S207). 

The estimated location information indicative of the position of the calibrating 
marker 106 to the image coordination system of the camera 110 is then produced by the 

25 estimated location information producing section 140 from the calibrating marker position 
information stored in the calibrating marker position information storing section 126 on the 
basis of the second optical position information produced by the second optical position 
information producing section 120 (in the step S208). The estimated location information 
produced by the estimated location information producing section 140 is stored in the 

30 estimated location information storing section 150 (in the step S209). 

The camera 110 and the recordable medium 193 are then transferred to the second 
working space from the first working space. The camera 110 is then mounted to the 
automotive vehicle 108 to be located at the predetermined position on the second 
coordination system 102 (in the step S210). 

35 The image information of the calibrating marker 106 is then obtained by the camera 

110 (in the step S211). As shown in FIG 15, the image location Pn' is searched by the 
image location searching section 272 in the estimated area 274 at the estimated location Pn 
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of the calibrating marker 106. The image location information is then extracted by the 
image location information extracting section 270. 

The calibration value to the second optical position information stored in the 
second optical position information storing section 130 is then calculated by the calibration 
5 value calculating section 280 on the basis of the image location information extracted by the 
image location information extracting section 270 and the estimated location information 
stored in the estimated location information storing section 150 (in the step S213). 

The second optical position information stored in the second optical position 
information storing section 130 is finally calibrated by the optical position information 
10 calibrating section 290 on the basis of the calibration value calculated by the calibration 
value calculating section 280 (in the step S214). In this embodiment, the above mentioned 
steps 101 to 115 may be executed as a program by a computer. 

From the above detailed description, it will be understood that the second 
embodiment of the camera calibrating apparatus 200 can easily extract the image location 
15 information of the calibrating marker 106 by reason that the image location of the 
calibrating marker 106 is searched by the image location searching section 272. 
(third embodiment) 

Referring now to the drawings, in particular to FIGS. 17 to 28, there is shown a 
third embodiment of a camera calibrating apparatus according to the present invention. 

20 The construction of the third embodiment of the camera calibrating apparatus 

according to the present invention will now be described hereinafter. 

As shown in FIGS. 17 to 19, the camera calibrating apparatus 300 is electrically 
connected to a camera 310 as an imaging device. The camera 310 includes a housing unit 
311 and an optical section 312 supported by the housing unit 311 to obtain image 

25 information through the optical section 312. 

The camera calibrating apparatus 300 comprises a first housing position 
information storing section 315 for storing first housing position information indicative of 
the position of the housing unit 311 to the first coordination system 301, a second housing 
position information storing section 316 for storing second housing position information 

30 indicative of the position of the housing unit 311 in the second coordination system 302, and 
a revising marker position information storing section 325 for storing revising marker 
position information indicative of the position of a revising marker 305 in the first 
coordination system 301. 

The first coordination system 301 is defined in a first working space such as a 

35 camera production plant to have X r axis, Y r axis, and Z r axis, while the revising marker 
305 laid out in the first working space causes the camera calibrating apparatus 300 to revise 
the camera 310 located in the first working space. The revising marker 305 is constituted 
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by a plurality of marking objects which are respectively located at predetermined positions 
on the first coordination system 301, and which are within a viewing field of the camera 310 
located in the first working space. 

The second coordination system 302 is defined in a second working space such as 
5 an automotive vehicle production plant. The second coordination system 302 has X2-axis, 
Y 2 -axis, and Z 2 -axis. The X 2 -axis and the Y 2 -axis collectively define X 2 -Y 2 plane to be 
represented by a road surface 302a on which an automotive vehicle 308 is located. The 
dividing marker 307 is located on a bumper 309 of the automotive vehicle 308. The 
dividing marker 307 is constituted by two marking objects which is within a viewing field 

10 of the camera 310 mounted on the automotive vehicle 308. The dividing marker 307 is 
located directly below the camera 310 mounted on the automotive vehicle 308. 

The camera calibrating apparatus 300 is adapted to revise the camera 310 in the 
first working space. The camera 310 is located at a predetermined position on the first 
coordination system 301, while the first housing position information storing section 315 is 

15 adapted to store first housing position information indicative of the position of the housing 
unit 311. Here, the term "the calibration of the camera 310" is intended to indicate the 
calculation of the accurate position of the optical section 312 of the camera 310 mounted on 
the automotive vehicle 308. 

The camera 310 revised by the camera calibrating apparatus 310 is mounted on the 

20 automotive vehicle 308 to be located at a predetermined position on the second coordination 
system 302 in the second working space, while the second housing position information 
storing section 316 is adapted to store second housing position information indicative of the 
position of the housing unit 311. Here, the second housing position information indicates 
an accurate position in which the housing unit 311 of the camera 310 is located when the 

25 camera 3 10 is mounted on the automotive vehicle 308. 

The dividing marker 307 is located on the bumper 309 of the automotive vehicle 
308 in spaced relationship with the position of the housing unit 311 represented by the 
second housing position information. This means that the dividing marker 307 is not 
moved with respect to the housing unit 311 when the automotive vehicle 308 is traveling on 

30 the road 302a. 

The camera calibrating apparatus 300 further comprises a first optical position 
information producing section 317 for producing first optical position information indicative 
of the position of the optical section 312 to the first coordination system 301, and a first 
optical position information storing section 318 for storing the first optical position 
35 information produced by the first optical position information producing section 317. 

The first optical position information producing section 317 is adapted to calculate 
the position of the optical section 312 to the first coordination system 301 from the revising 
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marker position information stored in the revising marker position information storing 
section 325 on the basis of the image information of the revising marker 305 obtained by the 
camera 310. Here, the term "the position of the optical section 312" is intended to indicate 
an optical center of the optical section 312 and the position of an optical axis, and other 
5 optical parameters. The calculation of the position of the optical section 312 to the first 
coordination system 301 is performed on the basis of the method disclosed in the document 
L 

The camera calibrating apparatus 300 further comprises a second optical position 
information producing section 320 for producing second optical information indicative of 

10 the position of the optical section 312 in the second coordination system 302, and a second 
optical position information storing section 330 for storing the second optical position 
information produced by the second optical position information producing section 320. 

The second optical position information producing section 320 is adapted to 
produce second optical position information indicative of the position of the optical section 

15 312 in the second coordination system 302 from the second housing position information 
stored in the second housing position information storing section 316 on the basis of the first 
housing position information stored by the first housing position information storing section 
315 and the first optical position information stored in the first optical position information 
storing section 318. 

20 The second optical position information producing section 320 is adapted to 

calculate the position of the optical section 312 in the second coordination system 302 on 
the basis of a method which is the same as the method of the second optical position 
information producing section 120 of the first embodiment. 

As shown in FIG 20, the camera 310 mounted on the automotive vehicle 308 

25 located in the second working space defines a camera coordination system 313 having 
x-axis, y-axis, and z-axis on the basis of the second optical position information. The 
origin point of the camera coordination system 313 is in register with the optical center of 
the optical section 312. The x-axis of the camera coordination system 313 extends in a 
horizontal direction of the camera 310. The y-axis of the camera coordination system 313 

30 extends in a vertical direction of the camera 310. The z-axis of the camera coordination 
system 313 is in axial alignment with the optical axis of the optical section 312. 

The image coordination system 314 is defined on a plane spaced apart from the 
origin point of the camera coordination system 313 with a focal length "f ' in the direction of 
the z-axis of the camera coordination system 313. The image coordination system 314 has 

35 p-axis and q-axis. The camera 310 is adapted to obtain, as image information, an optical 
image focused on the image coordination system 314 through the optical section 312. 

The camera calibrating apparatus 300 further comprises a calibrating section 360 
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for calibrating the second optical position information stored in the second optical position 
information storing section 330 on the basis of the motion vector of the image information 
obtained by the camera 310 in the second coordination system 302. 

The calibrating section 360 includes a plane-projected image producing section 361 
5 for producing a plane-projected image from the image information obtained by the camera 
310 on the second coordination system 302, a plane-projected image dividing section 362 
for dividing the plane-projected image produced by the plane-projected image producing 
section 361 into a plurality of image segments, a motion vector extracting section 363 for 
extracting the motion vector from the image segments divided by the plane-projected image 

10 dividing section362, a calibration value calculating section 364 for calculating a calibration 
value of the second optical position information stored by the second optical position 
information storing section 330 on the basis of the motion vector extracted by the motion 
vector extracting section 363, and an optical position information calibrating section 365 for 
calibrating the second optical position information stored by the second optical position 

15 information storing section 330 on the basis of the calibration value calculated by the 
calibration value calculating section 364. 

As shown in FIG 21, the plane-projected image produced by the plane-projected 
image producing section 361 is the sama as an image taken by an imaginary camera 370 
located on a plane which defines an image coordination system 371 parallel to the road 

20 surface 302a. The image of the road surface 302a focused on the image coordination 
system 371 is the same as the reduced-size road surface 302a. 

The plane-projected image producing section 361 is adapted to produce 
plane-projected image on the basis of a following method. 

In FIG 20, the angle of the x-axis of the image coordination system 313 to the road 

25 surface 302a is represented by a legend a . The angle of the y-axis of the image 
coordination system 313 to the road surface 302a is represented by a legend j3 . The 
distance between the origin point of the camera coordination system 313 and an intersecting 
point of the extension line of the z-axis of the camera coordination system 313 to the road 
surface 302a is represented by a legend "c". The X 2 -Y 2 surface of the second coordination 

30 system 302, i.e., the road surface 302a is represented by following equations (6) and (7). 

[b = sin p 

z = ax + by + c (7) 

35 
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The transformation to the camera coordination system 313 from the image 
coordination system 314 is represented by a following equation (8). Here, the legend (p, q) 
represents ordinate and abscissa of the image location P' to the image coordination system 
314. 
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10 In the equation (9), the translation vector "T" indicates a direction of the origin 

point of the second coordination system 302 to the origin point of the camera coordination 
system 313 and a distance between the origin point of the second coordination system 302 
and the origin point of the camera coordination system 313. The rotation matrix "R" 
represents a deviation between the second coordination system 302 and the camera 

15 coordination system 313 in the rotational direction. When the housing unit 311 is mounted 
on the slanted position on the automotive vehicle 308, the rotation matrix "R" is represented 
by the equations (3) to (5) shown in the first embodiment. Here, the legends Q , <£ , and 
cp represent the angle of the camera coordination system 313 to the x-axis of the camera 
coordination system 313, the angle of the camera coordination system 313 to the y-axis of 

20 the camera coordination system 313, and the angle of the camera coordination system 313 to 
the z-axis of the camera coordination system 313, respectively. 

When the housing unit 311 is mounted on the accurate position on the automotive 
vehicle 308, the plane-projected image producing section 361 is adapted to produce a 
plane-projected image shown in FIG 22(b) from the image information shown in FIG 22(a). 

25 In this plane-projected image, the lines 373 and 374 are in parallel relationship with each 
other by reason that the plane-projected image is the same as the image taken by the 
imaginary camera 370 located in parallel relationship with the road surface 302a. 

When the camera coordination system 313 is moved around the x-axis under the 
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condition that the housing unit 311 is downwardly slanted with respect to the automotive 
vehicle 308, the plane-projected image producing section 361 is adapted to produce a 
plane-projected image shown in FIG 23(b) from the image information shown in FIG 23(a). 

When the camera coordination system 313 is moved around the y-axis under the 
5 condition that the housing unit 311 is leftwardly panned with respect to the automotive 
vehicle 308, the plane-projected image producing section 361 is adapted to produce a 
plane-projected image shown in FIG 24(b) from the image information shown in FIG 24(a). 

When the camera coordination system 313 is moved around the z-axis under the 
condition that the housing unit 311 is leftwardly slanted with respect to the automotive 
10 vehicle 308, the plane-projected image producing section 361 is adapted to produce a 
plane-projected image shown in FIG 25(b) from the image information shown in FIG 25(a). 

As shown in FIGS. 22(b), 23(b), 24(b), and 25(b), the plane-projected image 
dividing section 362 is adapted to divide the plane-projected image into a plurality of image 
segments on the basis of the image information of the dividing markers 307 obtained by the 
15 camera 310. 

The plane-projected image dividing section 362 is adapted to superimpose dividing 
lines 383 and 384 on the plane-projected image on the basis of a base line 380 passing 
through the dividing markers 307. The dividing line 383 is in perpendicular relationship 
with the base line 380, and passes thought an intermediate point 381 defined between the 
20 dividing markers 307. On the other hand, the dividing line 384 is in parallel and spaced 
relationship with the base line 380. 

When the housing unit 311 is mounted on the slanted position of the automotive 
vehicle 308, more specifically, the camera coordination system 313 is moved around the 
z-axis, the base line 380 has a tilt to the right as will be seen from the plane-projected image 
25 shown in FIG 25(b). 

As shown in FIGS. 22(b), 23(b), 24(b), and 25(b), the motion vector extracting 
section 363 is adapted to extract the motion vectors 388a, 388b, 388c, and 388d from the 
four image segments 386a, 386b, 386c, and 386d divided by the plane-projected image 
dividing section 362. 

30 The directions of the motion vectors 388a to 388d are equal to one another when 

the automotive vehicle 308 is linearly traveling by reason that the motion vectors 388a to 
388d are calculated on the basis of the flow of the fragmental image of the image 
information obtained from the motion of the automotive vehicle 308, i.e., the motion of the 
camera 310 to the road. The following description will be directed to the motion vectors 

35 388a to 388d extracted from the image information obtained under the automotive vehicle 
308 is linearly traveling. 

When the housing unit 311 is mounted on the accurate position of the automotive 
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vehicle 308, the amplitudes of the motion vectors 388a, 388b, 388c, and 388d are equal to 
one another as shown in FIG 22(b). 

When! the housing unit 3 1 1 is mounted on the downwardly slanted position of the 
automotive vehicle 308, more specifically, the camera coordination system 313 is moved 
5 around the x-axis, the motion vectors 388a and 338b of the upper side fragmental image is 
larger than the motion vectors 388a and 338b of the lower side fragmental image as shown 
in FIG 23(b). 

When the housing unit 311 is mounted on the leftwardly slanted position of the 
automotive vehicle 308, more specifically, the camera coordination system 313 is moved 
10 around the y-axis, the motion vectors 388b and 338d of the right side fragmental image is 
larger than the motion vectors 388a and 338c of the left side fragmental image as shown in 
FIG 24(b). 

The calibration value calculating section 364 is adapted to produce a calibration 
value of the second optical position information on the basis of a following method. 

15 As shown in FIGS. 22(c), 23(c), and 24(c), the vector 390a equal in amplitude to 

the motion vector 388a, the vector 390b equal in amplitude to the motion vector 388b, the 
vector 390c equal in amplitude to the motion vector 388c, and the vector 390d equal in 
amplitude to the motion vector 388d are located on a composited vector calculating 
coordination system 391. 

20 Here, the vector 390a is located at the origin point of the composited vector 

calculating coordination system 391 in the direction of an upper-leftward angle of 45 
degrees. The vector 390b is located at the origin point of the composited vector 
calculating coordination system 391 in the direction of an upper-rightward angle of 45 
degrees. The vector 390c is located at the origin point of the composited vector calculating 

25 coordination system 391 in the direction of a lower-leftward angle of 45 degrees. The 
vector 390a is located at the origin point of the composited vector calculating coordination 
system 391 in the direction of a lower-rightward angle of 45 degrees. 

The composition of the vectors 390a to 390d is performed on the composited vector 
calculating coordination system 391. 

30 When the housing unit 311 is mounted on the accurate position of the automotive 

vehicle 308, the amplitude of the composited vector 392 is equal to zero by reason that the 
amplitudes of the motion vectors 388a, 388b, 388c, and 388d are equal to one another as 
shown in FIG 22(b). Here, each of the rotation angle 6 to the x-axis, the rotation angle 
(£> to the y-axis, and the rotation angle </> to the z-axis is equal to zero on the camera 

35 coordination system 313. 

When the housing unit 311 is mounted on the downwardly slanted position of the 
automotive vehicle 308, and more specifically, the camera coordination system 313 is 
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moved around the x-axis, the composited vector 392 points upward by reason that the 
vectors 390a and 390b are respectively larger in amplitude than the vectors 390a and 390b 
as shown in FIG 23(b). The rotation angle 8 to the x-axis is calculated on the basis of 
the equation (9) under the condition that the calculated rotation matrix "R" leads to the fact 
5 that the composited vector 392 is equal to zero. 

When the housing unit 311 is mounted on the leftwardly slanted position of the 
automotive vehicle 308, and more specifically, the camera coordination system 313 is 
moved around the y-axis, the composited vector 392 points rightward by reason that the 
vectors 390b and 390d are respectively larger in amplitude than the vectors 390a and 390c 

10 as shown in FIG 24(b). The rotation angle <£> to the y-axis is calculated on the basis of 
the equation (9) under the condition that the calculated rotation matrix "R" leads to the fact 
that the composited vector 392 is equal to zero. 

When, on the other hand, the housing unit 311 is mounted on the slanted position of 
the automotive vehicle 308, and more specifically, the camera coordination system 313 is 

15 moved around the z-axis, the rotation angle $ to the z-axis of the camera coordination 
system 313 is calculated on the basis of the slope of the base line 380 on the image 
coordination system 371 of the plane-projected image shown in FIG 25(b). 

As shown in FIG 26, the camera calibrating apparatus 300 thus constructed as 
previously mentioned can be realized by comprising a computer 391 for adjusting the 

20 camera 310, ECU 392 constituted as an image controlling apparatus for controlling the 
camera 310, and other elements. 

The computer 391 is shown in FIG 26(a) as including CPU, RAM, ROM, an 
input/output interface, and other units. The computer 391 is electrically connected to the 
camera 310 at the first working space. In this embodiment, the computer 391 constitutes 

25 each of the first housing position information storing section 315, the second housing 
position information storing section 316, the first optical position information producing 
section 317, the first optical position information storing section 318, the second optical 
position information producing section 320, the revising marker position information storing 
section 325, and the second optical position information storing section 330. 

30 The ECU 392 is shown in FIG 26(c) as including CPU, RAM, ROM, an 

input/output interface, and other elements. The ECU 392 is mounted on the automotive 
vehicle 308 to be electrically connected to the camera 310 in the second working space. In 
this embodiment, the ECU 392 constitutes each of the second optical position information 
storing section 330 and the calibrating section 360. 

35 As shown in FIG 26(b), the camera 310 is transferred to the second working space 

from the first working space with a recordable medium 393 such as for example CD-ROM 
and a magnetic disk. The recordable medium 393 has stored therein the second optical 
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position information to ensure that the second optical position information is effectively 
transferred to the ECU 392 from the computer 391. 

In this embodiment, the camera 310 and the recordable medium 393 are transferred 
to the second working space from the first working space, however, the camera 310, the 
5 second optical position information storing section 330, and the calibrating section 360 may 
be collectively transferred as a camera unit 194 to the second working space from the first 
working space as shown in FIG 27. 

The following description will be directed to the operation of the third embodiment 
of the camera calibrating apparatus according to the present invention. 

10 The second optical position information is calibrated by the camera calibrating 

apparatus 300 through the steps shown in the FIG 28. 

The camera 310 is firstly located at the predetermined position on the first 
coordination system 301 in the first working space (in the step S301). The first housing 
position information, the second housing position information, and the revising marker 

15 position information are then stored in the first housing position information storing section 
315, the second housing position information storing section 316, and the revising marker 
position information storing section 325, respectively (in the step S302). Here, the first 
housing position information, the second housing position information, and the revising 
marker position information are obtained on the basis of positions predetermined in the 

20 design process and measured by a measuring instrument. 

The image information of the revising marker 105 is then obtained by the camera 
310 (in the step S303). The first optical position information is then produced by the first 
optical position information producing section 317 from the revising marker position 
information stored in the revising marker position information storing section 325 on the 

25 basis of the image information of the revising marker 305 obtained by the camera 310 (in 
the step S304). The first optical position information produced by the first optical position 
information producing section 317 is then stored in the first optical position information 
storing section 318 (in the step S305). 

The second optical position information is then produced by the second optical 

30 position information producing section 320 from the second housing position information 
stored in the second housing position information storing section 316 on the basis of the first 
housing position information stored in the first housing position information storing section 
315 and the first optical position information stored in the first optical position information 
storing section 318 (in the step S306). The second optical position information produced 

35 by the second optical position information producing section 320 is then stored in the 
second optical position information storing section 330 (in the step S307). 

The camera 310 and the recordable medium 393 are then transferred to the second 
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working space from the first working space. The camera 310 is then mounted to the 
automotive vehicle 308 to be located at a predetermined position on the second coordination 
system 302 (in the step S308). 

The image of the road surface 302a is then taken by the camera 310 (in the step 
5 S309). As shown in FIGS. 22(b), 23(b), 24(b), and 25(b), the plane-projected image is 
produced from the image information obtained by the camera 310 on the second 
coordination system 302 (in the step S3 10). 

As shown in FIGS. 22(b), 23(b), 24(b), and 25(b), the plane-projected image 
produced by the plane-projected image producing section 361 is divided into a plurality of 
10 image segments 386a to 386d by the plane-projected image dividing section 362 on the 
basis of the image information of the dividing markers 307 obtained by the camera 310 (in 
the step S3 11). 

As shown in FIGS. 22(b), 23(b), 24(b), and 25(b), the motion vectors are extracted 
by the motion vector extracting section 363 from the image segments 386a to 386d divided 

15 by the plane-projected image dividing section 362 (in the step S3 12). 

As shown in FIGS. 25(b), the calibration value on z-axis of the second optical 
position information stored in the second optical position information storing section 330 is 
then calculated by the calibration value calculating section 365 on the basis of the slope of 
the base line 380 on the image coordination system 371 of the plane-projected image shown 

20 in FIG 25(b) (in the step S313). As shown in FIGS. 22(c), 23(c), and 24(c), the calibration 
value on each of x-axis and y-axis of the second optical position information stored in the 
second optical position information storing section 330 is then calculated by the calibration 
value calculating section 365 on the basis of the motion vectors extracted by the motion 
vector extracting section 363 (in the step S3 14). 

25 The second optical position information stored in the second optical position 

information storing section 330 is finally calibrated by the optical position information 
calibrating section 365 on the basis of the calibration value calculated by the calibration 
value calculating section 364 (in the step S3 15). In this embodiment, the above mentioned 
steps 301 to 315 may be executed as a program by a computer. 

30 From the above detailed description, it will be understood that the camera 

calibrating apparatus 300 can calibrate the second optical position information by using the 
motion vectors 388a to 388d. 

In this embodiment, the camera calibrating apparatus 300 can easily extract the 
motion vectors from the respective image segments 386a to 386d. 

35 In this embodiment, the camera calibrating apparatus 300 can accurately divide the 

plane-projected image by using the dividing marker 307. 
(fourth embodiment) 
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Referring now to the drawings, in particular to FIGS. 29 to 38, there is shown a 
fourth embodiment of a camera calibrating apparatus according to the present invention. 

The construction of the fourth embodiment of the camera calibrating apparatus 
according to the present invention will now be described hereinafter. 
5 As shown in FIGS. 29 to 31, the camera calibrating apparatus 400 is electrically 

connected to a camera 410 as an imaging device. The camera 410 includes a housing unit 
411 and an optical section 412 supported by the housing unit 411 to obtain image 
information through the optical section 412. 

The camera calibrating apparatus 400 comprises a first housing position 

10 information storing section 415 for storing first housing position information indicative of 
the position of the housing unit 411 to the first coordination system 401, a second housing 
position information storing section 416 for storing second housing position information 
indicative of the position of the housing unit 411 in the second coordination system 402, a 
revising marker position information storing section 425 for storing revising marker position 

15 information indicative of the position of a revising marker 405 to the first coordination 
system 401, and a vehicle position information storing section 426 for storing vehicle 
position information indicative of the position of, for example, a rear bumper 409 of an 
automotive vehicle 408 in the second coordination system 402. 

The first coordination system 401 is defined in a first working space such as a 

20 camera production plant to have X-axis, Yi-axis, and Z r axis. The revising marker 405 
laid out in the first working space causes the camera calibrating apparatus 400 is revise the 
camera 410 located in the first working space. The revising marker 405 is constituted by a 
plurality of marking objects which are respectively located at predetermined positions on 
the first coordination system 401, and which are within a viewing field of the camera 410 

25 located in the first working space. 

The second coordination system 402 is defined in a second working space such as 
an automotive vehicle production plant to have X 2 -axis, Y 2 -axis, and Z 2 -axis, while X 2 -axis 
and Y 2 -axis collectively define X 2 -Y 2 plane which is represented by a road surface 402a on 
which an automotive vehicle 408 is located. The calibrating plate 406 painted in a single 

30 color is located on the road surface 402a. The monochromatic calibrating plate 406 
located directly below the automotive vehicle 408 is different in brightness, chromaticity, 
and saturation of color from the automotive vehicle 408. 

The camera calibrating apparatus 400 is adapted to revise the camera 410 in the 
first working space. The camera 410 is located at a predetermined position on the first 

35 coordination system 401, while the first housing position information storing section 415 is 
adapted to store first housing position information indicative of the position of the housing 
unit 411. Here, the term "the calibration of the camera 410" is intended to indicate the 
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calculation of the accurate position of the optical section 412 of the camera 410 mounted on 
the automotive vehicle 408. 

The camera 410 revised by the camera calibrating apparatus 410 is mounted on the 
automotive vehicle 408 to be located at a predetermined position on the second coordination 
5 system 402 in the second working space, while the second housing position information 
storing section 416 is adapted to store second housing position information indicative of the 
position of the housing unit 411. Here, the second housing position information indicates 
an accurate position in which the housing unit 411 of the camera 410 is located when the 
camera 410 is mounted on the automotive vehicle 408. 

10 The camera calibrating apparatus 400 further comprises a first optical position 

information producing section 417 for producing first optical position information indicative 
of the position of the optical section 412 to the first coordination system 401, and a first 
optical position information storing section 418 for storing the first optical position 
information produced by the first optical position information producing section 417. 

15 The first optical position information producing section 417 is adapted to calculate 

the position of the optical section 412 to the first coordination system 401 from the revising 
marker position information stored in the revising marker position information storing 
section 425 on the basis of the image information of the revising marker 305 obtained by the 
camera 410. Here, the term "the position of the optical section 412" is intended to indicate 

20 an optical center of the optical section 412 and the position of an optical axis, and other 
optical parameters. The calculation of the position of the optical section 412 to the first 
coordination system 401 is performed on the basis of the method disclosed in the document 
1. 

The camera calibrating apparatus 400 further comprises a second optical position 
25 information producing section 420 for producing second optical information indicative of 
the position of the optical section 412 in the second coordination system 402, and a second 
optical position information storing section 430 for storing the second optical position 
information produced by the second optical position information producing section 420. 

The second optical position information producing section 420 is adapted to 
30 produce second optical position information indicative of the position of the optical section 
412 in the second coordination system 402 from the second housing position information 
stored in the second housing position information storing section 416 on the basis of the first 
housing position information stored by the first housing position information storing section 
415 and the first optical position information stored in the first optical position information 
3 5 storing section 418. 

The second optical position information producing section 420 is adapted to 
calculate the position of the optical section 412 in the second coordination system 402 on 
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the basis of a method which is the same as the method of the second optical position 
information producing section 120 of the first embodiment. 

As shown in FIG 32, the camera 410 mounted on the automotive vehicle 408 in the 
second working space defines a camera coordination system 413 having x-axis, y-axis, and 
5 z-axis on the basis of the second optical position information. The origin point of the 
camera coordination system 413 is in register with the optical center of the optical section 
412. The x-axis of the camera coordination system 413 extends in a horizontal direction of 
the camera 410. The y-axis of the camera coordination system 413 extends in a vertical 
direction of the camera 410. The z-axis of the camera coordination system 413 is in axial 

10 alignment with the optical axis of the optical section 412. 

The image coordination system 414 is defined on a plane spaced apart from the 
origin point of the camera coordination system 413 with a focal length "f 1 in the direction of 
the z-axis of the camera coordination system 413. The image coordination system 414 has 
p-axis and q-axis. The camera 410 is adapted to obtain, as image information, an optical 

15 image focused on the image coordination system 414 through the optical section 412. 

The camera calibrating apparatus 400 further comprises an estimated location 
information producing section 440 for producing estimated location information of the 
bumper 409 of the automotive vehicle 408 to the image coordination system 414 of the 
camera 410, and an estimated location information storing section 450 for storing the 

20 estimated location information produced by the estimated location information producing 
section 440. 

The estimated location information producing section 440 is adapted to produce 
estimated location information of the calibrating marker 406 to the image coordination 
system 414 of the camera 410 from the position information of the calibrating marker 406 
25 stored in the vehicle position information storing section 426 on the basis of the second 
optical position information produced by the second optical position information producing 
section 420. The calculation of the estimated location of the calibrating marker 406 to the 
image coordination system 414 of the camera 410 is performed on the basis of the above 
mentioned document 1 . 

30 As shown in FIG 33, the bumper 409 of the automotive vehicle 408 located on the 

second coordination system 402 is focused on the image location Pn f of the image 
coordination system 414 through the optical section 412. Here, the image location Pn' is in 
register with the estimated image location Pn calculated by the estimated image location 
calculating section 440 under the condition that the housing unit 411 is located on an 

35 accurate position of the automotive vehicle 408, more specifically, the housing unit 411 is 
located at the position represented by the second housing position information with no 
deviation to the accurate position. However, the housing unit 411, in general, is located at 
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an inaccurate position with deviation to the accurate position. This leads to the fact that 
the second optical position information is produced with deviation to the accurate position. 
As a result, the estimated location Pn is spaced apart from the image location Pn* of the 
image coordination system 414. 
5 The camera calibrating apparatus 400 further comprises a calibrating section 460 

for calibrating the second optical position information stored in the second optical position 
information storing section 430 to order to calibrate the above mentioned deviation of the 
second optical position information. 

The calibrating section 460 is adapted to calibrate the second optical position 
10 information stored in the second optical position information storing section 430 on the 
basis of the image information of the bumper 409 obtained by the camera 410 and the 
estimated location information stored in the estimated location information storing section 
450. 

The calibrating section 460 includes an image location information extracting 

15 section 470 for extracting image location information indicative of an image location of the 
bumper 409 to the image coordination system 414 of the camera 410 on the basis of the 
image information of the calibrating marker 406 obtained by the camera 110, a calibration 
value calculating section 480 for calculating a calibration value of the second optical 
position information stored in the second optical position information storing section 430 on 

20 the basis of the image location information extracted by the image location information 
extracting section 470 and the estimated location information stored in the estimated 
location information storing section 450, and an optical position information calibrating 
section 490 for calibrating the second optical position information stored in the second 
optical position information storing section 430 on the basis of the calibration value 

25 calculated by the calibration value calculating section 480. 

The calibration value calculating section 480 includes a superimposing section 481 
for superimposing one of a profile line of the bumper 409 of the image location information 
and a profile line of the bumper 409 of the estimated location information on the other of the 
profile line of the bumper 409 of the image location information and the profile line of the 

30 bumper 409 of the estimated location information, an extracting section 482 for extracting 
two or more points, for example points of end portions of the bumper, from the profile lines 
of the automotive vehicle superimposed by the superimposing section 481, and a calculating 
section 483 for calculating the calibration value of the second optical position information 
stored by the second optical position information by comparing the points of the image 

35 location information with the points of the estimated location information. 

The superimposing section 481 is adapted to superimpose the profile line " P " (see 
FIG 34(a)) of the bumper 409 of the estimated location information on the profile line " P' " 
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(see FIG 34(b)) of the bumper 409 of the image location information by allowing the profile 

line " P " of the bumper 409 of the estimated location information to be linearly and 

rotationally moved with respect to the profile line " P' " of the bumper 409 of the image 

location information (see FIG 34(c)). 
5 As shown in FIG 35(a), the extracting section 482 is adapted to extract two or more 

points (Pi, P 2 ) and (P,\ P 2 ) from the profile lines " P " and " P' " of the bumper 409 

superimposed by the superimposing section 481. 

The calculating section 483 is adapted to calculate a calibration value of the second 

optical position information on the basis of a method which is the same as the method of the 
10 calibration value calculating section 180 in the first embodiment by comparing the points 

(Pi, P 2 ) (see FIG 35(b)) of the estimated location information with the points (Pi\ P 2 f ) (see 

FIG 35(b)) of the image location information. 

As shown in FIG 36, the camera calibrating apparatus 400 thus constructed as 

previously mentioned can be realized by comprising a computer 491 for adjusting the 
15 camera 410, ECU 492 constituted as an image controlling apparatus for controlling the 

camera 410, and other elements. 

The computer 491 is shown in FIG 26(a) as including CPU, RAM, ROM, an 

input/output interface, and other elements. The computer 491 is electrically connected to 

the camera 410 at the first working space. In this embodiment, the computer 491 
20 constitutes each of the first housing position information storing section 415; the second 

housing position information storing section 416, the first optical position information 

producing section 417, the first optical position information storing section 418, the second 

optical position information producing section 420, the calibrating marker position 

information storing section 425, the calibrating marker position information storing section 
25 426, the second optical position storing section 430, the estimated location information 

producing section 440, and the estimated location information storing section 450. 

The ECU 492 is shown in FIG 36(c) as including CPU, RAM, ROM, an 

input/output interface, and other elements. The ECU 492 is mounted on the automotive 

vehicle 408 to be electrically connected to the camera 410 in the second working space. In 
30 this embodiment, the ECU 492 constitutes each of the second optical position information 

storing section 430, the estimated location information storing section 450, and the 

calibrating section 460. 

As shown in FIG 36(b), the camera 410 is transferred to the second working space 

from the first working space with a recordable medium 493 such as for example CD-ROM 
35 and a magnetic disk. The recordable medium 493 has stored therein each of the second 

optical position information and the estimated location information to ensure that the each 

of the second optical position information and the estimated location information is 
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effectively transferred to the ECU 492 from the computer 491. 

In this embodiment, the camera 410 and the recordable medium 493 are transferred 
to the second working space from the first working space, however, the camera 410, the 
second optical position information storing section 430, and the calibrating section 460 may 
5 be collectively transferred as a camera unit 494 to the second working space from the first 
working space as shown in FIG 37. 

The following description will be directed to the operation of the fourth 
embodiment of the camera calibrating apparatus according to the present invention. 

The second optical position information is calibrated by the camera calibrating 
10 apparatus 400 through the steps shown in the FIG 38. 

The camera 410 is firstly located at the predetermined position on the first 
coordination system 401 in the first working space (in the step S401). The first housing 
position information, the second housing position information, the revising marker position 
information, and the vehicle position information are then stored in the first housing 
15 position information storing section 415, the second housing position information storing 
section 416, the revising marker position information storing section 425, and the vehicle 
position information storing section 426, respectively (in the step S402). Here, the first 
housing position information, the second housing position information, the revising marker 
position information, and the vehicle position information are obtained on the basis of 
20 positions predetermined in the design process and measured by a measuring instrument. 

The image information of the revising marker 405 is then obtained by the camera 
410 (in the step S403). The first optical position information is then produced by the first 
optical position information producing section 417 from the revising marker position 
information stored in the revising marker position information storing section 425 on the 
25 basis of the image information of the revising marker 405 obtained by the camera 410 (in 
the step S404). The first optical position information produced by the first optical position 
information producing section 417 is then stored in the first optical position information 
storing section 418 (in the step S405). 

The second optical position information is then produced by the second optical 
30 position information producing section 420 from the second housing position information 
stored in the second housing position information storing section 416 on the basis of the first 
housing position information stored in the first housing position information storing section 
415 and the first optical position information stored in the first optical position information 
storing section 418 (in the step S406). The second optical position information produced 
35 by the second optical position information producing section 420 is then stored in the 
second optical position information storing section 430 (in the step S407). 

The estimated location information indicative of the position of the bumper 409 to 
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the image coordination system 414 of the camera 410 is then produced by the estimated 
location information producing section 440 from the vehicle position information stored in 
the vehicle position information storing section 426 on the basis of the second optical 
position information produced by the second optical position information producing section 
5 420 (in the step S408). The estimated location information produced by the estimated 
location information producing section 440 is stored in the estimated location information 
storing section 450 (in the step S409). 

The camera 410 and the recordable medium 493 are then transferred to the second 
working space from the first working space. The camera 410 is then mounted to the 

10 automotive vehicle 408 to be located at the predetermined position on the second 
coordination system 402 (in the step S410). 

The image of the bumper 409 is then taken by the camera 310 against the 
calibrating plate 406 (in the step S411). As shown in FIG 33, the image location 
information of the bumper 409 is extracted from the image information of the bumper 409 

15 obtained by the camera 410 on the image coordination system 414 of the camera 410 (in the 
step S412). 

As shown in FIG 34, the profile line F of the bumper 409 of the image location 
information is superimposed on the profile line P of the bumper 409 of the estimated image 
information by the superimposing section 481 (in the step S413). 

20 As shown in FIG 35, two or more points (Pi, P 2 ) and (Pi', P 2 ') of the overlapped 

portion are then extracted from the profile lines " P " and " P' " of the bumper 409 by the 
extracting section 482 (in the step S414). The calibration value of the second optical 
position information is then calculated by the calculating section 483 by comparing the 
points (Pi', P2) of the image location information with the points (Pi, P2) of the estimated 

25 location information (in the step S415). 

The second optical position information stored in the second optical position 
information storing section 430 is finally calibrated by the optical position information 
calibrating section 490 on the basis of the calibration value calculated by the calibration 
value calculating section 480 (in the step S416). In this embodiment, the above mentioned 

30 steps 401 to 415 may be executed as a program by a computer. 

From the above detailed description, it will be understood that the camera 
calibrating apparatus 400 can calibrate the second optical position information by using the 
motion vectors 388a to 388d. 

In this embodiment, the camera calibrating apparatus 400 can calibrate the second 

35 optical position information by using the bumper 409 forming part of the automotive vehicle 
408. 

In this embodiment, the camera calibrating apparatus 400 can accurately divide the 
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plane-projected image by using the dividing marker 307. 

As will be seen from the above detailed description, the camera calibrating 
apparatus according to the present invention can calibrate the optical parameter of the 
camera mounted to the automotive vehicle. 
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